Contactless-Type Sport Training Monitor Method

ABSTRACT

The present invention provides a contactless-type sport training monitor method, comprising: selecting at least an image database to recognize a plurality of expressions in the image database; making pre-processing for the plurality of expressions: using a convolutional neural network as a feature point extraction model; acquiring a human image; tracking a first target region and a second target region in the human image; making chrominance-based rPPG trace extraction; using the deep level model to compare the second target region image; and to calculate a post-exercise heart rate recovery achievement ratio, to judge the Rating of Perceived Exertion, and judging whether the human body is under overtraining status or not.

BACKGROUND OF THE INVENTION 1. Field of the Invention

The present invention relates to a contactless-type sport trainingmonitor method, particularly to a contactless-type sport trainingmonitor method by using an image method to monitor the heart rate andthe facial expression, in order to judge integratedly whether the humanbody is under overtraining status or not.

2. Description of the Prior Art

A successful training program aims is to increase athletes' orexercisers' fitness and endurance. This is achieved by the applicationof the appropriate training protocols for athletes to reach high peakperformance status during the competition season. However, overtrainingmay lead to fatigue, and limit athletes' performance. Sometimes thefatigue may even take the athletes several weeks to recover from. Atthis time, adjusting the training program becomes a very importanttopic.

Overtraining is characterized by imbalance between stress and recoveryafter exercising, and the syndrome is characterized by fatigue and lowperformance. At past, physicians, psychologists and physical therapistshave tried to reach diagnosis and consensus regarding overtrainingsyndrome. The overtraining can be diagnosed by physiological,psychological and behavioral, neuro endocrinal and biochemicalindicator, and immunological indicator etc.

The athletes' fitness and endurance should be considered during trainingprogram. Many clinical decision support systems can assist to monitorthe factors of physiological signal, emotion, or the level of fatigue ofthe athletes' by some wearable devices. In addition, the questionnaireshould also be taken into account to produce an assessment report. Suchprocess is cumbersome, and the results are not objective. Furthermore,the person may feel uncomfortable when wearing the devices during thetraining program.

As the abovementioned description, the people expect to estimate theRating of Perceived Exertion (RPE) without any wearable devices andquestionnaires. A camera based heart rate detection algorithm and afatigue expression feature extractor are fused to estimate thecontactless-type Rating of Perceived Exertion (RPE) value. The resultsshow that the heart rate detection algorithm and the fatigue heart ratedetection can obtain the similar results of wearable devices. It meansthe contactless-type sport training monitor method has certaincompetitiveness.

Heart rate and fatigue facial expression are regarded as the index ofovertraining. An image based overtraining monitoring system is proposedso that the training Rating of Perceived Exertion (RPE) level of theathlete can be estimated without any wearable devices. Since thequestionnaire is not considered, the results would more objective.

To address the abovementioned problems, it is necessary to develop anappropriate method to achieve contactless-type monitoring heart rate andfacial expression without any wearable devices attached on the athletes'body. This contactless-type method shall also be able to judge whetherthe athlete is under overtraining status or not by monitoring thefatigue, endurance and performance. The suitable measures shall beadopted to avoid the occurrence of overtraining status.

SUMMARY OF THE INVENTION

The embodiment of the present invention provides a contactless-typesport training monitor method, comprising: selecting at least an imagedatabase to recognize a plurality of expressions in the image database,respectively, in order to train a deep level model; makingpre-processing for the plurality of expressions, using a convolutionalneural network as a feature point extraction model in order to carry outthe feature fusion for the plurality of expressions, in order to extractan expression feature, according to the expression feature to analyzethe plurality of expressions as a facial expression image, and store inthe deep level model; acquiring a human image; tracking a first targetregion and a second target region in the specific human facial image,and acquiring a first target region image in the first target region anda second target region image in the second target region, respectively;making chrominance-based rPPG (CHROM rPPG) trace extraction for thefirst target region image, making Fourier transform for the first targetregion image; making Peak Selection for the first target region image,to obtain a heart rate value (HR); using the deep level, model tocompare the second target region image, to judge the second targetregion image as a specific facial expression image; and according to thespecific expression to judge the heart rate value is a resting heartrate or an exercising heart rate and a lowest heart rate, to calculate apost-exercise heart rate recovery achievement ratio (ΔHRR), according tothe post-exercise heart rate recovery achievement ratio to judge theRating of Perceived Exertion (RPE), and judging whether the human bodyis under overtraining status or not.

In the preferred embodiment, the image database is YouTube, RaFD orADFES.

In the preferred embodiment, training the deep level model includesusing the blurring, sharpening, lightened and darkening treatment stepto augment the plurality of expressions.

In the preferred embodiment, the dimension of the feature pointextraction model is 128.

In the preferred embodiment, the facial expression image is selectedfrom the group consisting of anger, disgust, fear, happiness, sadness orsurprise.

In the preferred embodiment, first target region image is the palm skinimage or face skin image of the human image.

In the preferred embodiment, the second target region image is thespecific facial expression image.

In the preferred embodiment, the Peak Selection steps comprise a motionnoise spectrum stability value is lower than a stability threshold, orthe difference between largest frequency and motion frequency is smallerthan a frequency threshold.

In the preferred embodiment, the stability threshold is −11 bpm, and thefrequency threshold is 4.5 bpm.

In the preferred embodiment, the motion noise spectrum stability valueis

${10{\log \left\lbrack \frac{E_{1}}{E_{2}} \right\rbrack}},$

wherein, E₁ is the largest frequency energy, and E₂ is the total signalenergy minus E₁.

In the preferred embodiment, the post-exercise heart rate recoveryachievement ratio is

$\frac{{E\left\lbrack {HR}_{ex} \right\rbrack} - {HRR}_{lowest}}{{E\left\lbrack {HR}_{ex} \right\rbrack} - {E\left\lbrack {HR}_{rest} \right\rbrack}},$

wherein, HRR_(lowest) is the lowest heart rate value, E[HR_(ex)] is theaverage exercising heart rate value, and HR_(rest) is the averageresting heart rate value.

In order to further understand the features and technological content ofthe present invention, please refer to the following detaileddescription and attached figures of the present invention. Nevertheless,the attached figures are used for reference and description, which arenot used for limiting the present invention.

BRIEF DESCRIPTION OF THE DRAWINGS

The foregoing aspects and many of the attendant advantages of thisinvention will become more readily appreciated as the same becomesbetter understood by reference to the following detailed description,when taken in conjunction with the accompanying drawings, wherein:

FIG. 1 illustrates a flowchart of contactless-type sport trainingmonitor method of the present invention;

FIG. 2 illustrates a contactless-type sport training monitor device ofthe present invention;

FIG. 3A illustrates the heart rate spectrum with high spectrum stabilityof the present invention; and

FIG. 3B illustrates the heart rate spectrum with low spectrum stabilityof the present invention.

DESCRIPTION OF THE PREFERRED EMBODIMENT

In the following context, the specific embodiments are used to describethe contactless-type sport training monitor method of the presentinvention. The people who are familiar to this art can understand theadvantages and efficacies of the present invention easily from thecontent disclosed in this article. The present invention can also beimplemented or applied by other different embodiments. Every detail inthis article can also be modified and changed based on differentviewpoints and applications without violating the spirit of the presentinvention. In addition, the figures in the present invention are onlybrief description, and they are not drawn in actual dimension to reflectthe actual size. The following description of preferred embodimentdescribes the viewpoint of the present invention in more detail, whichwill not limit the scope of the present invention by any viewpoint.

Please refer to FIG. 1 and FIG. 2. FIG. 1 illustrates a flowchart ofcontactless-type sport training monitor method of the present invention.FIG. 2 illustrates a contactless-type sport training monitor device ofthe present invention. The contactless-type sport training monitormethod of the present invention shown in FIG. 1 comprises the followingsteps:

Firstly, as shown in Step S102 of FIG. 1, selecting an image database204 to recognize a plurality of expressions in the image database 204,respectively, in order to train a facial expression feature extractionmodel. It has to describe that the image database is YouTube, RaFD orADFES. The image database is used to train a facial expression featureextraction model. The videos are downloaded from the image database,which are cropped with the time interval set to 1, 2 or 3 seconds toproduce the facial images. The face images would be filtered out if itis non-face, two-face, or too blurred, or too dark, or the expressionswhich are hard to be identified. In addition, because the environmentcondition of YouTube image database is quite bad, the RaFD or ADFESimage database is added to train the image database of deep level modelin the embodiment. The training images of deep level model would beaugmented by blurring, sharpening, lightened and darkening.

Please also refer to FIG. 3. FIG. 3 illustrates the Neighbor-centerdifference image method of the present invention. As shown in FIG. 3, itis able to increase the edge efficiency of the plurality of firstexpression P1.

As shown in Step S104 of FIG. 1 and also refer to FIG. 2, aconvolutional neural network feature paint is used to extract model 208to obtain an expression feature from expression P. According to theexpression feature, analyze the plurality of expressions as a facialexpression image, and store in the facial expression feature extractionmodel 206 in FIG. 2. As an example in FIG. 3, when the feature pointextraction model 208 analyze and judge the expression feature extractedfrom expression P as a happy facial expression image, the happy facialexpression image is stored in the facial expression feature extractionmodel 206. It has to note that the dimension of feature point extractionmodel is 128. The facial expression image is selected from the groupconsisting of anger, disgust, fear, happiness, sadness or surprise.

Still as shown in Step S106 of FIG. 1, the image device 202 of FIG. 2 isused to extract the human image I of human body 220.

As shown in Step S108 of FIG. 1, tracking the first target region T1 andthe second target region T2 in the human image I, and acquiring thefirst target region image I1 of the first target region T1, and thesecond target region image I2 of the second target region T2respectively, as shown in FIG. 2. It has to note that the first targetregion image I1 is the palm skin image or facial skin image of humanimage I. The second target region image I2 is the specific facialexpression image, as shown in FIG. 2.

As shown in Step S110 of FIG. 1, making the chrominance-based rPPG(CHROM rPPG) trace extraction for the first target region image I1. TheCHROM rPPG trace extraction is passing an assumed standard skin color toenhance the robustness of motions and eliminate the noise generated inthe motion (such as swing) domain.

As shown in Step S112 of FIG. 1, making Fast Fourier Transform (FFT) forthe first target region image I1 to transform to the domain.

As shown in Step S114 of FIG. 1, making Peak Selection for the firsttarget region image I1, to obtain a heart rate value HR. The PeakSelection is used to reduce the motion noise, so a Spectrum Stability(SS) is defined as:

${{SS} = {10{\log \left\lbrack \frac{E_{1}}{E_{2}} \right\rbrack}}},$

Wherein, E₁ is the energy of the largest frequency, and E₂ is the totalsignal energy minus E₁.

As shown in the embodiment of FIG. 1, the Peak Selection Step includesthe spectrum stability value of motion noise is lower than a stabilitythreshold T_(h), or the difference between largest frequency f_(h) andmotion frequency f_(ex) smaller than a frequency threshold T_(f).

As shown in the embodiment of FIG. 1, the bandwidth of stabilitythreshold T_(h) is −11 bpm, and the bandwidth of frequency thresholdT_(f) is 4.5 bpm.

Please refer to FIG. 3A and FIG. 3B. FIG. 3A illustrates the heart ratespectrum with high spectrum stability of the present invention. FIG. 3Billustrates the heart rate spectrum with low spectrum stability of thepresent invention.

As shown in Step S116 of FIG. 1, the deep level model is used to comparethe second target region image I2, to judge the second target regionimage I2 is a specific expression.

As shown in Step S118 of FIG. 1, according to the specific expression tojudge whether the heart rate value HR is resting heart rate HR_(rest),or exercising heart rate HR_(ex), and lowest heart rate HRR_(lowest), tocalculate the post-exercise heart rate recovery achievement ratio ΔHRR.According to the post-exercise heart rate recovery achievement ratioΔHRR to judge the Rating of Perceived Exertion (RPE), and judge whetherthe human body is under the overtraining status or not. Thepost-exercise heart rate recovery achievement ratio ΔHRR is defined as:

${\Delta \; {HRR}} = \frac{{E\left\lbrack {HR}_{ex} \right\rbrack} - {HRR}_{lowest}}{{E\left\lbrack {HR}_{ex} \right\rbrack} - {E\left\lbrack {HR}_{rest} \right\rbrack}}$

Wherein, HRR_(lowest) is the lowest heart rate value, E[HR_(ex)] is theaverage exercising heart rate value, HR_(rest) is the average restingheart rate value.

By the invention, the people can estimate the Rating of PerceivedExertion (RPE), i.e., a camera based fatigue heart rate detectionalgorithm and a fatigue facial expression image feature extractor arefused to estimate the contactless-type Rating of Perceived Exertion(RPE) value. Therefore, the invention operating the fatigue heart ratedetection, and a fatigue facial expression image feature extractor canobtain the similar results of wearable devices and the questionnaires.

Indeed, the invention can achieve the contactless-type monitoring heartrate and the facial expression image without any wearable devicesattached on the athletes' body. Therefore, the invention, which is thecontactless-type method exactly can judge whether the athlete is underovertraining status or not.

It is understood that various other modifications will be apparent toand can be readily made by those skilled in the art without departingfrom the scope and spirit of the invention. Accordingly, it is notintended that the scope of the claims appended hereto be limited to thedescription as set forth herein, but rather that the claims be construedas encompassing all the features of patentable novelty that reside inthe present invention, including all features that would be treated asequivalents thereof by those skilled in the art to which the inventionpertains.

What is claimed is:
 1. A contactless-type sport training monitor method,comprising: selecting at least an image database to recognize aplurality of expressions in an image database, respectively, in order totrain a facial expression feature extraction model; acquiring a sequenceof human images; tracking a first target region and a second targetregion in said human image, and acquiring a first target region imagesin said first target region and a second target region image in saidsecond target region, respectively; making a rPPG trace extraction forsaid first target region images: making a time-frequency transform forsaid first target region images; using the signal processing for saidfirst target region image, to obtain some physiological signals; usingsaid facial expression feature extraction model to extract the facialexpression features on the said second target region images according tosaid facial features and said physiological signals to judge an indexfor judging whether said human body is under overtraining status or not.2. The contactless-type sport training monitor method according to claim1, wherein the image database can be YouTube, RaFD and ADFES.
 3. Thecontactless-type sport training monitor method according to claim 1,wherein the facial expression feature extraction model can be aconvolutional neural network.
 4. The contactless-type sport trainingmonitor method according to claim 1, wherein the facial expressionfeature extraction model can be trained by the facial images withspecific facial expressions.
 5. The contactless-type sport trainingmonitor method according to claim 4, wherein the facial expression canbe the basic facial expression consisting of anger, disgust, fear,happiness, sadness, surprise, and contempt.
 6. The contactless-typesport training monitor method according to claim 4, wherein the facialexpression can be the action unit coding system.
 7. The contactless-typesport training monitor method according to claim 4, wherein the facialexpression can be the index of valence and arousal.
 8. Thecontactless-type sport training monitor method according to claim 1,wherein the first target region image comprises the palm skin image orface skin image of the human image.
 9. The contactless-type sporttraining monitor method according to claim 1, wherein the second targetregion image comprises the facial expression image.
 10. Thecontactless-type sport training monitor method according to claim 1,wherein the signal processing step comprises a motion noise spectrumstability value is lower than a stability threshold, or the differencebetween largest frequency and motion frequency is smaller than afrequency threshold.
 11. The contactless-type sport training monitormethod according to claim 1, wherein the stability threshold comprises−11 bpm, and the frequency threshold is 4.5 bpm.
 12. Thecontactless-type sport training monitor method according to claim 9,wherein the motion noise spectrum stability value comprises${10{\log \left\lbrack \frac{E_{1}}{E_{2}} \right\rbrack}},$ whereinE1 is the largest frequency energy, and E2 is the total signal energyminus E1.
 13. The contactless-type sport training monitor methodaccording to claim 1, wherein the physiological signals comprises aheart rate, a heart rate recovery.
 14. The contactless-type sporttraining monitor method according to claim 12, post-exercise heart raterecovery achievement ratio comprises$\frac{{E\left\lbrack {HR}_{ex} \right\rbrack} - {HRR}_{lowest}}{{E\left\lbrack {HR}_{ex} \right\rbrack} - {E\left\lbrack {HR}_{rest} \right\rbrack}},$wherein HRR_(lowest) comprises the lowest heart rate value, E[HR_(ex)]comprises the average exercising heart rate value and HR_(rest)comprises the average resting heart rate value.